Standard Extended Strings (Topic)

Cartography Artisan

So i'm making extended strings a thing, for comments or generally unused-in-game strings in CHKDraft, and if any other programs wanted to support these they'll need to know these standards. If others want support for their extended string formats, they'll need to share their format as well (if someone has done so elsewhere, lmk).

To do so, i'm making a new section titled KSTR, which will be ignored by starcraft and most editors, and removed by TiniMap and other (good) compressors. The format of KSTR is as follows:

Title: KSTR
Size: at least 4 bytes
Format:
��VersionNum - 4 bytes
��(the rest of the format depends on VersionNum, see below)

Version 2

Version 1 - Expired (DON'T USE)

Post has been edited 14 time(s), last time on Mar 11 2014, 5:39 pm by jjf28.

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

"If your program changes numStrs, it must change all kString ids currently in other sections of the map to match this change, or all existing KSTRs will become corrupted."

This bit is interesting. Is there anyway for you to design this so that doesn't happen? You're talking about changing the maximum number of strings here, instead of the 'current' number of strings, right? Even so, what if when they changed numStrs, there was a byte in your header which was for numStrs, would that allow the rest of the map to keep the same numbers for KSTR without having to change them?

So if numStrs changed from 1024 to 2048, your editor could check the value of numStrs in the KSTR header, see "oh, it's 2048 now" and therefore only lookup strings that were 2049+ from KSTR and use STR to look up anything from 1025-2048? I guess if they had a KSTR at position 1512, and changed numStrs, then unless they changed all references to 1512 to something 2048+, the editor would start looking in STR instead of KSTR to find this entry.

So could KSTR numbering be completely different from regular string numbering? Like start at 32,769 instead of 1025?

None.

Cartography Artisan

yes numStrings = max strings in the STR section

I suppose they could count down from 32767 (or something), my reason for starting at the end of STR was because I haven't yet tested this in programs like SCMDraft, i was thinking if it came across, say, string 1045 for a comment, it might overflow and crash, and the lower it was, the less chance it would actually leave the programs memory bound and actually crash

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Not sure how SC/utilities handle it, but if it works I think map extensions are most elegantly implemented as separate file(s) in the MPQ, then have the program check for the existence of the file(s) in an MPQ to determine that it's extended. This could also be used for appending source code for map compilers and the like.

None.

👻 👾 👽 💪

I think they work just fine as separate CHK sections, since that is how the map format works anyway.

Also any good editor should make sure that a string is valid only if it is not higher than the string count, but I am interested to see what happens. Also when I first started reading the post I was thinking negative string IDs would be neat.

EDIT:
In looking, I guess I myself never bothered with invalid string IDs in tinymap2. xD But that's an easy fix.

TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
$\:farty\:$ This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

Cartography Artisan

Quote

I think map extensions are most elegantly implemented as separate file(s) in the MPQ

Four little things detract from that

- Potentially unhanded hash collisions (I don't know if blizzard kept a list of hashes and just made sure new file names didn't collide or if they actually just said "screw it, prolly won't happen", which is really bad if you actually run the numbers)
- Metadata couldn't be stored in standalone chks (sometimes useful if you edit it than put it back into an MPQ, though this should depreciate in usefulness as chkdraft gets further along)
- Metadata is not with the chk file, which could be confusing for some code implementations (for example, the chk file could be a child of another class, and that would be where you're editing strings from, requiring you to break OOP rules to get the metadata back to another file in the MPQ, or they only load and deal with the scenario file, and would require reorganization to load such metadata (neither of which are problems for current chkdraft/chkdraft based code, but would have been in the past code versions))
- Metadata's already stored in the chk file, like ISOM (as farty ninja'd)

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Dunno what a hash collision is, but maps contain .wav's with arbitrary contents and arbitrary file names, dunno why it would discriminate against other arbitrary files with arbitrary file names.

Dunno how significant this concern is; wav's couldn't be stored in standalone chk's.

Discussions of design and implementation should be mutually exclusive.

Standard data is stored in the Blizzard-defined CHK. User-defined data would be elegantly defined in separate files; it helps with the modularity. Might as well keep the pristine CHK and define an entirely new file, for less conceptual clutter.

Dunno how much it matters in the end, as both methods do work. Just saying what I'd do if I was making something like this.

None.

Cartography Artisan

they're all tiny concerns, just giving reasons why i'd want to stick with it this way, rather than flat out ignoring your suggestion (a pet peeve of mine when other people do it)

Quote

Discussions of design and implementation should be mutually exclusive.

In a normal coding community i'd agree, but in such a small one I consider aiding the coding process to be important, an argument which could work for both methods come to think of it.

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

👻 👾 👽 💪

Hash collisions aren't an issue with the MPQ format.

User-defined data works perfectly well inside a CHK. Any non-standard sections are ignored (as is the case with every editor-only section). Any editor that doesn't recognize the sections will (usually) just discard them. I suppose a separate file has a lower chance of being discarded, but you can't rely on that.

TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
$\:farty\:$ This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

Cartography Artisan

Quote

Any editor that doesn't recognize the sections will (usually) just discard them.

oo, now that could be reason enough to store as a separate file... i'll see how many old programs do that.

If we were crazy enough, it could be stored as a WAV used by an impossible condition trigger

(not really, it should be destructible by compressors)

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

👻 👾 👽 💪

Quote from

jjf28

If we were crazy enough, it could be stored as a WAV used by an impossible condition trigger

(not really, it should be destructible by compressors)

TinyMap2 would remove it.

WAVs in "Never" triggers are not counted as being used.

TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
$\:farty\:$ This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

Quote from

jjf28

Quote

Any editor that doesn't recognize the sections will (usually) just discard them.

oo, now that could be reason enough to store as a separate file... i'll see how many old programs do that.

But if those same programs throw away sections they don't understand/use, then it's quite likely if you opened a map with extra files they didn't understand, they'd throw them away too. It is worth investigating though.

None.

Cartography Artisan

Quote

But if those same programs throw away sections they don't understand/use, then it's quite likely if you opened a map with extra files they didn't understand, they'd throw them away too. It is worth investigating though.

Depends, how I had originally wrote load/save chk was to go through the the file and copy sections i'd use into buffers, then in writing the save method I'd call each buffer by name to write to the file, then overwrite the original chk with the new one, so I wasn't deliberately discarding sections, but they were lost none the less, if anyone else was doing something like that it would cause problems.

Perhaps in programs where the author forgot WAVs existed they might create a new MPQ file every time and insert a scenario file, but I think it's more likely that they'd just replace the scenario file in the existing MPQ file to avoid losing WAVs or having to temporarily pull each WAV out. Ripping files out from an MPQ otherwise requires a deliberate effort to delete them, rather than an oversight in not preserving unrecognized sections.

Quote

impossible condition trigger

chose these specific words for a reason (conditions to the effect of, but not using never) :kame:

ne ways I wouldn't go that far to stubbornly force my metadata on the world

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

👻 👾 👽 💪

Quote from

Lanthanide

Quote from

jjf28

Quote

Any editor that doesn't recognize the sections will (usually) just discard them.

oo, now that could be reason enough to store as a separate file... i'll see how many old programs do that.

But if those same programs throw away sections they don't understand/use, then it's quite likely if you opened a map with extra files they didn't understand, they'd throw them away too. It is worth investigating though.

It's harder to remove files than to remove sections. Overwriting the entire CHK, as jff described, is how they are generally "discarded". You'd have to deliberately remove files in order for them to be gone. But I'm still not sure you can rely on programs not doing that.

TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
$\:farty\:$ This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!

What does each member of propStruct mean? Is they ChkDraft-exclusive thing?

EUD

Cartography Artisan

red green and blue define a color that the text is to display in, isUsed tells you whether the extended string is disabled or w/e (so you'd just return a null string insted), has priority tells you that this string is more important than one without this property, so a comment displays first or w/e

bold
underlined, and
italics are font effects
sizes are the size your text displays in

the prop struct will probably only be used by chkdraft, but it's there for anyone if anyone wants to use it

Quote from name:self'

I suppose they could count down from 32767 (or something)

just thought i'd mention why this value would be my second choice rather than negative numbers, some places strings are designated as 4 bytes and some places they are 2 bytes, between those being unsigned or signed and being converted to shorts and what have you there's a lot of room for making stupid mistakes, making counting down from 32767 easier the less crazy making choice.

Post has been edited 1 time(s), last time on Sep 16 2013, 5:30 pm by jjf28.

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

Cartography Artisan

For general strings scmd auto-increases numStrings, so a new standard will be needed.

Did a bit of documentation on how sc and scmd handle extended strings for this purpose.

Code

string usage/existing extended string handling

MRGN
string per location (2-bytes)
SC: Ignored
SCMD: main display, main tree, classic trigs, text trigs -- replaced with new string upon load

TRIG
trigger actions: text (4-bytes)
SC: NUL string used
SCMD: classic trigs, text trigs -- crashes on load if > 0x7FFFFFFF, otherwise treats as invalid

MBRF
trigger actions: text (4-bytes)
SC: NUL string used
SCMD: classic briefing, text briefing -- crashes on load if > 0x7FFFFFFF, otherwise treats as invalid

SPRP
scenario name (2-bytes)
SC: Uses file name insted
SCMD: map properties -- often crashes on load
scenario description (2-bytes)
SC: NUL string used
SCMD: map properties -- often crashes on load

FORC
string for force names (2-bytes)
SC: NUL string used
SCMD: forces, text trigs, text briefing -- treats as invalid

WAV
string per .wav (4-bytes)
SC: Ignored if unused
SCMD: sound editor, classic trigs, classic briefing, text trigs, text briefing -- crashes on load if > 0x7FFFFFFF, otherwise treats as invalid

UNIS
string for unit names (2-bytes)
SC: Ignored if unused
SCMD: main tree, status bar, unit properties, unit settings, classic trigs, classic briefing -- always uses UNIx if avaliable?

SWNM
string for switches (4-bytes)
SC: Ignored if unused
SCMD: switch manager, classic trigs, text trigs -- uses default switch string for display, cannot select switch from most lists

UNIx
string for unit names (2-bytes)
SC: Ignored if unused
SCMD: main tree, status bar, unit properties, unit settings, classic trigs, classic briefing) -- uses default unit string

SC summary: has absolutely no problems with extended strings

SCMD summary: extended strings should not be used in SPRP, and should never be greater than 0x7FFFFFFF, if switches must be editable in SCMD, then don't use them in SWNM, if your map will be frequently opened with SCMD, then don't use them in MRGN

So in v2 (will be written up soon), extended strings will count down from 65535 and emphasis will be put on treating string numbers as unsigned

I'm assuming no one used v1 just yet so i'ma scratch it off so no one implements it, if you did use it, it shouldn't be too hard to change your code to work in v2

SCMD does not delete extra sections, so it will be kept as an extra section for now.

Post has been edited 7 time(s), last time on Sep 22 2013, 4:15 pm by jjf28.

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

SDE, BWAPI owner, hacker.

For extended comment strings, could you use a value of -1 for the STR index and just use an unused 32-bit field to refer to an index in the KSTR section, rather than combining them?

>> BWAPI <<

Cartography Artisan

-1 for the comment string is off-limits as far as i'm concerned as that would crash scmdraft on load (but 0x7FFFFFFF would work)

Regardless, I don't wish to use an unused field because:
��- These don't exist in all sections that may use extended strings (a separate section exclusively for comment strings may be appropriate, will build if people don't think 65535 strings is enough for their map&commenting needs)
��- These may be randomly cleared or reused by other old programs (perhaps they had the same idea using unused fields for meta-data, which I hear some programmers did)
��- I may or may not already be using these fields for my trigger groups feature :kame:

(depending on how other programs handle such data)

Edit (3? or something): I suppose new sections like xMRG could be made for sections like MRGN that are of static size to store their string index and other meta-data, eliminating the first concern, &I have ~(28-8)*7 unused bits in the players part of TRIG for trigger groups, if I were to store that data with triggers

Post has been edited 3 time(s), last time on Sep 22 2013, 5:00 pm by jjf28.

TheNitesWhoSay - Clan Aura - github

Reached the top of StarCraft theory crafting 2:12 AM CST, August 2nd, 2014.

👻 👾 👽 💪

Would the extended strings really be used for all of that? SC sure can't use them (unless you are doing something fancy that I missed ...). You could instead count up from 65536 (use 0x10000 as a flag, maybe) for any that are 4-bytes and just put a table somewhere with the extended IDs for locations and whatever else has two bytes.

TinyMap2 - Latest in map compression! ( 7/09/14 - New build! )
EUD Action Enabler - Lightweight EUD/EPD support! (ChaosLauncher/MPQDraft support!)
EUDDB - topic - Help out by adding your EUDs! Or Submit reference files in the References tab!
MapSketch - New image->map generator!
EUDTrig - topic - Quickly and easily convert offsets to EUDs! (extended players supported)
SC2 Map Texture Mask Importer/Exporter - Edit texture placement in an image editor!
$\:farty\:$ This page has been viewed [img]http://farty1billion.dyndns.org/Clicky.php?img.gif[/img] times!